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Abstract 

We consider the empirical risk minimization problem for linear supervised learning, with regular- 
ization by structured sparsity-inducing norms. These are defined as sums of Euclidean norms on 
certain subsets of variables, extending the usual £i-norm and the group ^i-norm by allowing the 
subsets to overlap. This leads to a specific set of allowed nonzero patterns for the solutions of such 
problems. We first explore the relationship between the groups defining the norm and the resul- 
ting nonzero patterns, providing both forward and backward algorithms to go back and forth from 
groups to patterns. This allows the design of norms adapted to specific prior knowledge expressed 
in terms of nonzero patterns. We also present an efficient active set algorithm, and analyze the 
consistency of variable selection for least-squares linear regression in low and high-dimensional 
settings. 

Keywords: sparsity, consistency, variable selection, convex optimization, active set algorithm 



1. Introduction 



Sparse linear models have emerged as a powerful framework to deal with various supervised estima- 
tion tasks, in machine learning as well as in statistics and signal processing. These models basically 
seek to predict an output by linearly combining only a small subset of the features describing the 
data. To simultaneously address this variable selection and the linear model estimation, £i-norm 
regularizati o n has become a popul a r tool, that benefi ts both from efficient algorithms (see, e.g., 
Efron et al.l. |2004j; iLee et al.l. 120071 : lYuan et al.l. |2009j, and multiple references ther ein) and well- 



developed theory for generaUzation properties and varia ble selection consistency (|Zhao and Yu . 
20061 : IWainwrighj-boogUeickel et al.l.l200^ : lzhang| . l2009h . 



When regularizing by the £i-norm, sparsity is yielded by treating each variable individually, 
regardless of its position in the input feature vector, so that existing relationships and structures 
between the variables (e.g., spatial, hierarchical or related to the physics of the problem at hand) 
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are merely disregarded. However, many practical situations could benefit from this type of prior 
knowledge, potentially both for interpretability purposes and for improved predictive performance. 

For instance, in neuroimaging, one is interested in localizing areas in functional magnetic res- 
onance imaging (fMRI) or magnetoenc ephalography (MEG) signals that are discrirninatiy e to dis- 
tinguish between different brain states ( Gramfort and Kowalski . 2009 : Xiang et al. . 20091 . and ref- 
erences therein). More precisely, fMRI responses consist in voxels whose three-dimensional spatial 
aiTangement respects the anatomy of the brain. The discrirn inative voxels are thus expected to have 
a specific localized spatial organization ( Xiang et al. , 20091) . which is important for the subsequent 
identification task performed by neuroscientists. In this case, regularizing by a plain ^i-norm to 
deal with the ill-conditionedness of the problem (typically only a few fMRI responses described 
by tens of thousands of voxels) would ignore this spatial configuration, with a potential loss in 
interpretability and performance. 

Similarly, in face recognition, robustness to occlusions can be inc reased by considerin g as fea- 
tures, sets of pixels that form small convex regions on the face images (IJenatton et al.U2010r). Again. 



a plai n £i-norm regularization fails to encode this specific spatial locality constraint (IJenatton et al 



20I0f ). Still in comput er vision, object and s cene recog nition generally se ek to extract bounding 



boxes in either images (IHarzallah et al.l |2009|) or videos (IDalal et al.l 120061) . These boxes concen- 
trate the predictive power associated with the considered object/scene class, and have to be found by 
respecting the spatial arrangement of the pixels over the images. In videos, where series of frames 
ai^e studied over time, the temporal coherence also has to be taken into account. An unstructured 
sparsity-inducing penalty that would disregard this spatial and temporal information is therefore not 
adapted to select such boxes. 

Another example of the need for higher-order prior knowledge comes from bioinformatics. In- 
deed, for the diagnosis of tumors, the profiles of array -based compara t ive ge nomic hybridization 



(arrayCGH) can be used as inputs to feed a classifier (IRapaport et al.l . 120081) . These profiles ai^e 



characterized by plenty of variables, but only a few samples of such profiles are available, prompt- 
ing the need for variable selection. Because of the specific spatial organization of bacterial artificial 
chromosomes along the genome, the set of discriminative features is expected to have specific con- 
tiguous patterns. Using this prior knowl edge on top of a standard sparsity-inducing method leads to 
improvement in classification accuracy ( Rapaport et al. , 20081) . In the context of multi-task regres- 
sion, a genetic problem of interest is to find a mapping between a small subset of si ngle nucleotide 
polymorphisms (SNP's) that have a phenotypic impact on a given family of genes (IKim and Xingl. 
12009. ) . This tai^get family of genes has its own structure, where some genes share c ommon genetic 
chara cteristics, so that these genes can be embedded into a underlying hierarchy (IKim and Xingl. 



20091 ). Exploiting directly this hierarchical information in the regularization term outperforms the 
unstructured approach with a standard £^ -n orm. Such hierarchical structures have been likewise 
useful in the context of wavelet regression (|Zhao et al.i. i2009^) or kernel-based non linear" variable 
selection (lBachll2009bl) . 

These real world examples motivate the need for the design of sparsity-inducing regulariza- 
tion schemes, capable of encoding more sophisticated prior knowledge about the expected sparsity 
patterns. 

As mentioned above, the ^i-norm focuses only on cardinality and cannot easily specify side 
information about the patterns of nonzero coefficients ("no nzero patterns") indu c ed in the solution, 
since they are all theore t ically possible. Group -norms ( Yuan and Lin . 20061 : Roth and Fischer . 



20081 : iHuang and Zhangl. l2009l) consider a partition of all variables into a certain number of subsets 
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and penalize the sum of the Euclidean norms of each one, leading to selection of groups rather 
than individual variables. Moreover, recent works have considere d overlapping but nested groups 
in constrain ed situations such as trees and directed acyclic graphs (|Zhao et all |2009|; iBachl. l2009bl : 
Kim and X ing. 200^). 



In this paper, we consider all possible sets of groups and characterize exactly what type of 
prior knowledge can be encoded by considering sums of norms of overlapping groups of variables. 
Before describing how to go from groups to nonzero patterns (or equivalently zero patterns), we 
show that it is possible to "reverse-engineer" a given set of nonzero patterns, i.e., to build the unique 
minimal set of groups that will generate these patterns. This allows the automatic design of sparsity- 
inducing norms, adapted to target sparsity patterns. We give in Section [3] some interesting examples 
of such designs in specific geometric and structured configurations, which covers the type of prior 
knowledge available in the real world applications described previously. 

As will be shown in Section |3l for each set of groups, a notion of hull of a nonzero pattern 
may be naturally defined. For example, in the particular case of the two-dimensional planar grid 
considered in this paper, this hull is exactly the axis-aligned bounding box or the regular- convex 
hull. We show that, in our framework, the allowed nonzero patterns are exactly those equal to their 
hull, and that the hull of the relevant variables is consistently estimated under certain conditions, 
both in low and high-dimensional settings. Moreover, we present in Section H] an efficient active set 
algorithm that scales well to high dimensions. Finally, we illustrate in Section [6]the behavior of our 
norms with synthetic examples on specific geometric settings, such as lines and two-dimensional 
grids. 



Notation. For x G and q G [1, oo), we denote by ||x||g its ^^-norm defined as {Y^^=i \xj\'^y^'i 
and \\x\\^ = maxjgji p} \xj\. Given w e W and a subset J of {1, ... ,p} with cardinality | J|, 
wj denotes the vector in rI'^I of elements of w indexed by J. Similarly, for a matrix M £ M^^™, 
M/j G rI^I^^I-'I denotes the sub-matrix of M reduced to the rows indexed by / and the columns 
indexed by J. For any finite set A with cardinality \A\, we also define the |^|-tuple {y"')a£A G 
]gpx|A| jj^g collection of p-dimensional vectors y'^ indexed by the elements of A. Furthermore, 
for two vectors x and y in W, we denote hy x o y = {xiyi, . . . , Xpyp)~^ G W the elementwise 
product of X and y. 



2. Regularized Risk Minimization 

We consider the problem of predicting a random variable y G 3^ from a (potentially non random) 
vector X G M^, where y is the set of responses, typically a subset of M. We assume that we are 
given n observations {xi,yi) £ W x y, i = 1, . . . ,n. We define the empirical risk of a loading 
vector ID G M*' as L{w) = ^ X^i^i '^"''^i)' where £ : 3^ x M i— )• M+ is a loss function. We 
assume that I is convex and continuously differentiable with respect to the second parameter. Typical 
examples of loss functions are the square loss for least squares regression, i.e., £{y, y) = ^{y — yY 
with y G M, and the logistic loss y) = log(l + e~^^) for logistic regression, with y G {—1, 1}. 

We focus on a general family of sparsity-inducing norms that allow the penalization of subsets 
of variables grouped together. Let us denote by ^ a subset of the power set of {1, . . . ,p} such 
that y^Q^gG = {!,... i.e., a spanning set of subsets of {!,... Note that Q does not 
necessarily define a partition of {1, ... and therefore, it is possible for elements ofQ to overlap. 
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We consider the norm U defined by 



\Wn 



o w\ 



2 ' 



> if j G G and df 



(1) 







where {dP)Geg is a |^|-tuple of p-dimensional vectors such that 
otherwise. A same variable Wj belonging to two different groups Gi , G2 G ^ is allowed to be 
weighted differentiy in Gi and G2 (by respectively and dj^). We do not study the more general 
setting where each dP would be a (non-diagonal) positive-definite matrix, which we defer to future 
work. Note that a lai^ger family of penalties with similar propert ies may be obtained by replacing the 
^2-norm in Eq. ^ by other ^^-norm, q > 1 ( Zhao et all 20091) . Moreover, non-convex alternatives 



to Eq. ([T]) with quasi-norms in place of norms m ay also be interesting, in order to yield sparsity 
more aggressively (see, e.g. Jjenatton et al.U2010h . 

This general formulation has several important sub-cases that we present below, the goal of 
this paper being to go beyond these, and to consider norms capable to incorporate richer prior 
knowledge. 

• ^2 -norm: Q is composed of one element, the full set {1, ... 



^i-norm: Q is the set of all singletons, leading to the Lasso (ITibshiranil 1 1 9961) for the square 
loss. 

^2-norm and ^i-norm: Q is the set of all singlet ons and the full set |1 , . . . ,p}, leading (up 
to the squaring of the ^2 -norm) to the Elastic net dZou and Hastiel l2005h for the square loss. 



Gro up ^1 -norm: Q is an y partition of {1, ... leading to the group-Lasso for the square 
loss ( Yuan and Lin . 2006h . 



Hierarchical norms: when the set {1, . . . , p ) is ernbedded into a tree ( Zhao et al. , 2009h or 
more generally into a directed acyclic graph (IBachl l2009bl) . then a set of p groups, each of 
them composed of descendants of a given variable, is considered. 



We study the following regularized problem: 



mm 

w&w n 



1 " 



(2) 



1=1 



where /x > is a regularization parameter. Note that a non-regularized constant term could be 
included in this formulation, but it is left out for simplicity. We denote by w any solution of 
Eq. ( B. Regularizing b y linear combinations of (non-squared) ^2-norms is known to induce sparsity 
in w (IZhao et al.ll2009l) : our grouping leads to specific patterns that we describe in the next section. 



3. Groups and Sparsity Patterns 

We now study the relationship between the norm Q defined in Eq. ([T]) and the nonzero patterns the 
estimated vector w is allowed to have. We first characterize the set of nonzero patterns, then we 
provide forward and backward procedures to go back and forth from groups to patterns. 
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Figure 1 : Groups and induced nonzero pattern: three sparsity -inducing groups (middle and right, 
denoted by {Gi, G2, G3}) with the associated nonzero pattern which is the complement 
of the union of groups, i.e., (Gi U G2 U G3Y (left, in black). 



3.1 Stable Patterns Generated by G 



The regularization term Q{w) = X^Gee IM'' o ^^112 is a mixed (^1, £2)-norm (IZhao et al.L 120091) . At 
the group level, it behaves like an £i-norm and therefore, Q induces group sparsity. In other words, 
each d'^ o w, and equivalently each wg (since the support of d*^ is exactly G), is encouraged to 
go to zero. On the other hand, within the groups G G ^, the ^2-iiorm does not promote sparsity. 
Intuitively, for a certain subset of groups G' '^G, the vectors wg associated with the groups GgG' 
will be exactly equal to zero, leading to a set of zeros which is the union of these groups, {Jq^qiG. 
Thus, the set of allowed zero patterns should be the union-closure of G, i.e. (see Figure [T] for an 
example): 



Z=( [j G;G' ^g\. 



(3) 

Geg' ' 

The situation is however slightly more subtle as some zeros can be created by chance (just as reg- 
ularizing by the ^2-norm may lead, though it is unlikely, to some zeros). Nevertheless, Theorem |2] 
shows that, under mild conditions, the previous intuition about the set of zero patterns is con^ect. 
Note that instead of considering the set of zero patterns Z, it is also convenient to manipulate 
nonzero patterns, and we define 



V = l f) G'; G'<^g\ = {Z'; ZeZ}. 



(4) 



Gee' 

We can equivalently use V or Z by taking the complement of each element of these sets. 

The following two results chai^acterize the solutions of the problem Q. We first gives suf- 
ficient conditions under which this problem has a unique solution. We then formally prove the 
aforementioned intuition about the zero patterns of the solutions of namely they should belong 
to Z. In the following two results (see proofs in Appendix |A] and Appendix |B]|, we assume that 
i : {y, y') 1— l{y, y') is nonnegative, twice continuously differentiable with positive second deriva- 
tive with respect to the second variable and non-vanishing mixed derivative, i.e., for any y, y' in M, 
|^(y,y')>Oand^(y,y')/0. 

Proposition 1 Let Q denote the Gram matrix ^ XixJ. We consider the optimization problem 
in Eq. (jS]) with fi > 0. If Q is invertible or if {1, ... ,p} belongs to G, then the problem in Eq. ^ 
admits a unique solution. 
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Note that the invertibiUty of the matrix Q requires p < n. For high-dimensional settings, the 
uniqueness of the solution will hold when {1, . . . ,p} belongs to Q, or as further discussed at the end 
of the proof, as soon as for any j, A; G {!,••• ,p}, there exists a group G ^ Q which contains both 
j and k. Adding the group {1, . . . ,p} to Q will in general not modify V (and Z), but it will cause 
Q to lose its minimality (in a sense introduced in the next subsection). Furthermore, adding the full 
group {1, . . . ,p} has t o be put in parallel wit h the equivalent (up to the squaring) ^2-norm term in 
the elastic-net penalty (|Zou and HastieL |2005|), whose effect is to notably ensure strong convexity. 
Fo r more sophist i cated uniqueness conditions t hat we have not exp lored here, we re fer the r eaders 



to 



Osborne et all (|2000l. Theorem 1, 4 and 5), iRosset et all (|2004l. Theorem 5) or bossall jlOOl 



Theorem 3) in the Lasso case, and iRoth and Fischerl (l2008h for the group Lasso setting. We now 
turn to the result about the zero patterns of the solution of the problem in Eq. (|2l): 



Theorem 2 Assume that Y = {yi, . . . ,yn)~^ is a realization of an absolutely continuous probability 
distribution. Let k be the maximal number such that any k rows of the matrix (xi, . . . , x„) € W^^ 
are linearly independent. For ^ > 0, any solution of the problem in Eq. ([2]) with at most k — 1 
nonzero coefficients has a zero pattern in Z = {Ucec?' — <^i^ost surely. 

In other words, when Y = (yi, . . . , yn)^ is a realization of an absolutely continuous probability 
distribution, the sparse solutions have a zero pattern inZ = {IJgg6' S' ^ Q} almost surely. As 
a corollary of our two results, if the Gram matrix Q is invertible, the problem in Eq. Q has a unique 
solution, whose zero pattern belongs to Z almost surely. Note that with the assumption made on Y, 
Theorem|2]is not directly applicable to the classification setting. Based on these previous results, we 
can look at the following usual special cases from Section|2](we give more examples in Section [331) : 

• the set of allowed nonzero patterns is composed of the empty set and the full set 
{l,...,p}. 

• £i-norm: V is the set of all possible subsets. 

• ^2-norm and £i-norm: V is also the set of all possible subsets. 

• Group £i-norm: V = Z is the set of all possible unions of the elements of the partition 
defining Q. 



Hierarchical norms: t he set of patte rns V is then all sets J for which all ancestors of elements 
in J are included in J (|Bachl.l2009bh . 



Two natural questions now arise: (1) starting from the groups Q, is there an efficient way to gen- 
erate the set of nonzero patterns V; (2) conversely, and more importantly, given V, how can the 
groups Q — and hence the norm ^(w) — be designed? 



3.2 General Properties of G, Z and V 

We now study the different properties of the set of groups Q and its corresponding sets of patterns 
Z and V. 
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Figure 2: ^-adapted hull: the pattern of variables / (left and middle, in red) and its hull (left and 
right, hatched square) that is defined by the complement of the union of groups that do 
not intersect /, i.e., (d U G2 U G^if. 



Closedness. The set of zero patterns Z (respectively, the set of nonzero patterns V) is closed under 
union (respectively, intersection), that is, for all i(' G N and aW zi, . . . , zk £ U^i ^fc ^ ^ 
(respectively, pi, . . . ,pk G V, 0^=1 Pfc ^ This implies that when "reverse-engineering" the 
set of nonzero patterns, we have to assume it is closed under intersection. Otherwise, the best we 
can do is to deal with its intersection-closure. For instance, if we consider a sequence (see Figure^, 
we cannot take V to be the set of contiguous patterns with length two, since the intersection of such 
two patterns may result in a singleton (that does not belong to V). 



Minimality. If a group in Q is the union of other groups, it may be removed from Q without 
changing the sets Z or V. This is the main argument behind the pruning backward algorithm in 
Section [331 Moreover, this leads to the notion of a minimal set Q of groups, which is such that for 
all Q' Q Z whose union-closure spans Z, we have Q C Q' . The existence and uniquenes s of a 
minimal set is a consequence of classical results in set theory (|Doignon and FalmagneLll998h . The 
elements of this minimal set are usually referred to as the atoms of Z. 

Minimal sets of groups are attractive in our setting because they lead to a smaller number of 
groups and lower computational complexity — for example, for 2 dimensional-grids with rectangular 
patterns, we have a quadratic possible number of rectangles, i.e., \Z\ = 0{p'^), that can be generated 
by a minimal set Q whose size is |^| = 0{^). 



Hull. Given a set of groups Q, we can define for any subset / C {1, . . . , ^5} the Q -adapted hull, or 
simply hull, as: 



Hull(/) 



Gee, Gni 



U 



which is the smallest set in V containing / (see Figure |2ll; we always have / C Hull(/) with 
equality if and only if / G "P. The hull has a clear geometrical interpretation for specific sets Q 
of groups. For instance, if the set Q is formed by all vertical and horizontal half-spaces when the 
variables are organized in a 2 dimensional-grid (see Figure [5]), the hull of a subset / C {1, . . . ,p} 
is simply the axis-aligned bounding box of /. Similarly, when Q is the set of all half-spaces with all 
possible orientations (e.g., orientations ib7r/4 are shown in Figure IQ, the hull becomes the regular 



7 



Jenatton, Audibert and Bach 




Figure 3: The DAG for the set Z associated with the 2 x 2-grid. The members of Z are the com- 
plement of the areas hatched in black. The elements of Q (i.e., the atoms of Z) are 
highlighted by bold edges. 



convex hulfl- Note that those interpretations of the hull are possible and valid only when we have 
geometrical information at hand about the set of variables. 

Graphs of patterns. W e consider the directed acyclic graph (DAG) stemming from the Hasse 



diagram dCameronl . 1 1994) of the partially ordered set (poset) D). By definition, the nodes of 
this graph are the elements G of ^ and there is a directed edge frorn Gi to G2 if and only if 
Gi D G2 and there exists no G € ^ such that Gi D G D G2 ( Cameroiil. 1994 ). We can also build 
the corresponding DAG for the set of zero patterns Z Q, which is a super-DAG of the DAG of 
groups (see Figure [3] for examples). Note that we obtain also the isomorphic DAG for the nonzero 
patterns V, although it corresponds to the poset (P, c): this DAG will be used in the active set 
algorithm presented in Section m 

Prior works with nested groups (ZhaoetaL, 20091 : Bachl. 2009bl : Kim and Xing . 2009h have 
also used a similar DAG structure, with the slight difference that in these works, the corresponding 
hierarchy of v ariables is built fro m the prior knowledge about the probl e m at ha nd (e.g., the tree 
of wavelets in IZhao et al.l ( 2009 ). the decomposition of kernels in iBachI (|2009bl ') or the hierarchy 
of genes in lKim and XingI (120090 ). The DAG we introduce here on the set of groups naturally and 
always comes up, with no assumption on the variables themselves (for which no DAG is defined in 
general). 



3.3 From Patterns to Groups 

We now assume that we want to impose a priori knowledge on the sparsity structure of a solution w 
of our regularized problem in Eq. This information can be exploited by restricting the patterns 
allowed by the norm Q.. Namely, from an intersection-closed set of zero patterns Z, we can build 
back a minimal set of groups Q by iteratively pruning away in the DAG corresponding to Z, all 
sets which are unions of their parents. See Algorithm [U This algorithm can be found under a 

1 . We u se the term convex informally here. It can however be made precise with the notion of convex subgraphs Ich una . 
[l997h . 
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different form in iDoignon and Falmagnd (119981) — we present it through a pruning algorithm on the 
DAG, which is natural in our context (the proof of the minimality of the procedure can be found in 
AppendixO. The complexity of Algorithm[T]is 0{p\Z\'^). The pruning may reduce significantly the 
number of groups necessary to generate the whole set of zero patterns, sometimes from exponential 
in p to polynomial in p (e.g., the ^i-norm). In Section [331 we give other examples of interest where 
1^1 (and 1^1) is also polynomial in p. 



Algorithm 1 Backward procedure 

Input: Intersection-closed family of nonzero patterns V. 
Output: Set of groups Q. 

Initialization: Compute Z = {P'^; P € V} and set Q = Z. 
Build the Hasse diagram for the poset {Z,d). 
for t = miiiGez \ G\to max^e^ |G| do 
for each node G ^ Z such that |G| = t do 

if (UcGChiidren(G) C = then 
if (Parents(G) / 0) then 

Connect children of G to parents of G. 
end if 

Remove G from Q. 
end if 
end for 
end for 



Algorithm 2 Forward procedure 

Input: Set of groups Q = {Gi, . . . , Gm}- 
Output: Collection of zero patterns Z and nonzero patterns V. 
Initialization: Z = {0}. 
for m = 1 to M do 
C = {0] 

for each Z G Z do 

if {Gm ^ Z) and (VG G {d, . . . , G^-i}, G C Z U G„ ^ G C then 

G^GU{ZUG„J. 
end if 
end for 
Z^ Z\JC. 
end for 

V = {Z^; Z G Z]. 



3.4 From Groups to Patterns 



The forward procedure presented in Algorithm|2l taken from lDoignon and Falmagnd (119981) . allows 
the construction of Z from Q. It iteratively builds the collection of patterns by taking unions, 
and has complexity 0(p|2||^p). The general scheme is straightforward. Namely, by considering 
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increasingly larger sub-families of Q and the collection of patterns already obtained, all possible 
unions are formed. However, some attention needs to be paid while checking we are not generating 
a pattern already encountered. Such a verification is performed by the if condition within the inner 
loop of the algorithm. Indeed, we do not have to scan the whole collection of patterns already 
obtained (whose size can be exponential in \Q\), but we rather use the fact that Q generates Z. Note 
that in general, it is not possible to upper bound the size of \Z\ by a polynomial term in p, even 
when Q is very small (indeed, \Z\ = TP and \Q\ = p for the £i-norm). 



3.5 Examples 

We now present several examples of sets of groups Q, especially suited to encode geometric and 
temporal prior information. 

Sequences. Given p variables organized in a sequence, if we want only contiguous nonzero pat- 
terns, the backward algorithm will lead to the set of groups which are intervals [1, p„i} and 
[fc)P]fcg{2,...,p}' with both \Z\ = 0{p^) and \Q\ = 0{p) (see Figure |4l). Imposing the contiguity 
of the nonz ero patterns is for inst ance relevant for the diagnosis of tumors, based on the profiles of 



arrayCGH dRapaport et al.l.l2008h . 




Figure 4: (Left) The set of blue groups to penahze in order to select contiguous patterns in a 
sequence. (Right) In red, an example of such a nonzero pattern with its corresponding 
zero pattern (hatched area). 



Two-dimensional grids. In Section [6j we notably consider for V the set of all rectangles in two 
dimensions, leading by the previous algorithm to the set of axis-aligned half-spaces for Q (see 
Figure [5]l, with \Z\ = 0{p'^) and \Q\ = 0(y/p). This type of structure is encountered in object or 
scene recognition, where the selected rectangle would correspond to a certain box inside an im age, 
that concentrates the predictive power for a given class of object/scene (|Harzallah et al.U2009l ). 

Larger set of convex patterns can be obtained by adding in Q half -planes with other orientations 
than vertical and horizontal. For instance, if we use planes with angles that are multiples of 7r/4, 
the nonzero patterns of V can have polygonal shapes with up to 8 faces. In this sense, if we keep on 
adding half-planes with finer orientations, the nonzero patterns of V can be described by polygonal 
shapes with an increasingly larger number of faces. The standai^d notion of convexity def ined in 

w ould correspond to the situation where an infinite number of orientations is considered dSoilld. 
l2003h . See Figure [6l The number of groups is linear in ^ with constant growing linearly with 
the number of angles, while \Z\ grows more rapidly (typically non-polynomially in the number of 
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angles). Imposing such convex-like regions turns out to be useful in computer vision. For instance, 
in face reco gnition, it enables th e design of localized features that improve upon the robustness to 



occlusions ( Jenatton et al. . 2010h . 




Figure 5: Vertical and horizontal groups: (Left) the set of blue and green groups with their (not dis- 
played) complements to penalize in order to select rectangles. (Right) In red, an example 
of nonzero pattern recovered in this setting, with its corresponding zero pattern (hatched 
area). 




Figure 6: Groups with ib7r/4 orientations: (Left) the set of blue and green groups with their (not 
displayed) complements to penalize in order to select diamond-shaped patterns. (Right) 
In red, an example of nonzero pattern recovered in this setting, with its corresponding 
zero pattern (hatched area). 



Extensions. The sets of groups presented above can be straightforwardly extended to more com- 
plicated topologies, such as three-dimensional spaces discretized in cubes or spherical volumes 
discretized in slices. Similar properties hold for such settings. For instance, if all the axis-aligned 
half-spaces ai^e considered for ^ in a three-dimensional space, then V is the set of all possible 
rectangular boxes with \V\ = 0{p^) and \Q\ = 0{p^^^). Such three-dimensional structures 
may be interesting to retrieve discriminative and local sets of voxels from fMRI/MEEG responses 
( Gramfort and Kowalskil . l2009l : IXiang et al.Ll2009il Moreover, while the two-dimen sional rectangu- 
lar pa tterns described previously are adapted to find bounding boxes in static images (IHarzallah et al. , 



scene recognition in videos requires to deal with a third temporal dimension (iDalal et al 
20061). This may be achieved by designing appropriate sets of groups, embedded in the three- 



dimensional space obtained by tracking the frames over time. 
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Representation and computation of G. The sets of groups described so far can actually be rep- 
resented in a same form, that lends itself well to the analysis of the next section. When dealing with 
a discrete sequence of length I (see Figure H]), we have 

g = {g'l; ke{l,...,l-l}}U{gi; ke{2,...,l}}, 

= Gmi U bright, 

with 5^ = {z; 1 < i < k} and g'^ = {i; I > i > k}. In other words, the set of groups Q can be 
rewritten as a partitior|^in two sets of nested groups, ^left and bright- 

The same goes for a two-dimensional grid, with dimensions h x / (see Figure |5] and Figure |6l). 
In this case, the nested groups we consider are defined based on the following groups of variables 

= {{iJ) e{l,...,l}x{l,...,h}; cos{e)i + sm{e)j < k}, 

where /c G Z is taken in an appropriate range. 

The nested groups we obtain in this way are therefore parameterized by an angl^ 9, 9 £ 
(— 7r;7r]. We refer to this angle as an orientation, since it defines the normal vector to 

the line S M?]cos{9)i + sui{9)j = k}. In the example of the rectangular groups (see Fig- 

ureO, we have four orientations, with 9 G {0, 7r/2, — 7r/2, vr}. More generally, if we denote by © 
the set of the orientations, we have 

Q=[jQe, 

where G indexes the partition of Q in sets Qq of nested groups of variables. Although we have 
not detailed the case of M?, we hkewise end up with a similar partition of Q. 



4. Optimization and Active Set Algorithm 



For moderate values of p, one may obtain a solution for Eq. Q using generic toolbo xes for second- 

order cone programming (SOCP) whose time complexity is equal to 0{p^-^+\Q\^'^) (|Boyd and Vandenberghd . 
2004j), which is not appropriate when p or \Q\ are lai^ge. This time complexity corresponds to the 
computation of Eq. Q for a single value of the regularization parameter fi. 

We present in this section an active set algorithm (Algorithm [S]) that finds a solution for Eq. Q 
by considering increasingly larger active sets and checking global optimality at each step. When the 
rectangular groups are used, the total complexity of this method is in 0{s max{p^' s^'^}), where 
s is the size of the active set at the end of the optimization. Here, the sparsity prior is exploited for 
computational advantages. Our active set algorithm needs an underlying black-box SOCP solver; 
in this paper, we consider both a f irst order approa c h (see Append i x IHl) and a SOCP methoc]^ — in 



our experiments, we use SDPT3 (|Toh et al 



19991 : iTiituncii et al.L l2003r) . Our active set algorithm 



extends to general overlapping groups the work of BachI ( 2009b ). by further assuming that it is 



computationally possible to have a time complexity polynomial in the number of variables p. 

We primarily focus here on finding an efficient active set algorithm; we defer to futu r e wor k the 
design of specific SOCP solvers, e.g., based on proximal techniques (see, e.g.. iTsengl . |2009|. and 
numerous references therein), adapted to such non-smooth sparsity-inducing penalties. 



2. Note the subtlety: the sets Qg are disjoint, that is Qe n Qg' = for 6 7^ 9' , but groups in Qe and Qgi can overlap. 

3. Due to the discrete nature of the underlying geometric structure of Q, angles 6 that are not multiple of 7r/4 (i.e., such 
that tan(S) ^ Z) are dealt with by rounding operations. 

4. The CH — h/Matlab code used in the experiments may be downloaded from the authors website. 
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4.1 Optimality Conditions: from Reduced Problems to Full Problems 

It is simpler to derive the algorithm for the following regularized optimization proble ni which 
has the same solution set a s the regularized problem of Eq. ^ when fi and A are allowed to vary 
( Borwein and Lewis , 20061 . see Section 3.2): 



1 " A 
mm-y^£{yi,w^x^) + -[n{w)f. (5) 

i=l 



In active set methods, the set of nonzero variables, denoted by J, is built incrementally, and the 
problem is solved only for this reduced set of variables, adding the constraint wjc = to Eq. 
In the subsequent analysis, we will use arguments based on duality to monitor the optimality of 
our active set algorithm. We denote by L{w) = ^ J2^=i ^{Vi^w^ ^i) the empirical risk (which is 
by assumption convex and continuously differentiable) and by L* its Fenchel-conjugate, defined as 
( Boyd and Vandenberghe . 2004; Borwein and Lewis! 2006): 



L*{u)= svlY) {w^ u — L{w)} . 



The restriction of L to ]RI"'I is denoted Lj{wj) = L{w) for wj = wj and wjc = 0, with Fenchel- 
conjugate Lj. Note that, as opposed to L, we do not have in general Lj{kj) = L*{k) for hj = kj 
and kjc = 0. 

For a potential active set J C { 1 , . . . , p} which belongs to the set of allowed nonzero patterns V, 
we denote by Qj the set of active groups, i.e., the set of groups G ^ Q such that G n J / 0. We 
consider the reduced norm Qj defined on M'"'! as 

^Jiwj) = ^\\dj owj\\2= \\d'jowj\\2, 
Geg GeGj 

and its dual norm = maxQj(^„^)<i w] also defined on RI-^L The next proposition (see 

proof in Appendix iDl) gives the optimization problem dual to the reduced problem (Eq. (O below): 

Proposition 3 (Dual Problems) Let J C {1, . . . The following two problems 

A 2 

mill Lj{wj) + - [VLj{wj)] , (6) 

max -L:j{-Kj)-^[n*j{Kj)\\ (7) 

are dual to each other and strong duality holds. The pair of primal-dual variables {wj,kj} is 
optimal if and only if we have 

\kj =-VLj{wj), 

[w-Jkj ={[n:j{r,j)f = x[nj{wj)f. 



5. It is also possible to derive the active set algorithm for the constrained formulation 
min^gRP ^ ^(j/i, w^a;i) such that < A. However, we empirically found it more difficult to 

select A in this latter formulation. 
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As a brief reminder, the duality gap of a minimization problem is defined as the difference be- 
tween the primal and dual objec tive functions, evaluated for a feasible pair of primal/dual variables 
(|Boyd and Vandenberghd . l2004j. see Section 5.5). This gap serves as a certificate of (sub)optimality: 
if it is equal to zero, then the optimum is reached, a nd provided that strong duality holds, the con- 
verse is true as well (|Boyd and Vandenberghd . l2004l see Section 5.5). 

The previous proposition enables us to derive the duality gap for the optimization problem 
Eq. that is reduced to the active set of variables J. In practice, this duality gap will always 
vanish (up to the precision of the underlying SOCP solver), since we will sequentially solve Eq. ^ 
for increasingly larger active sets J. We now study how, starting from the optimality of the problem 
in Eq. we can control the optimality, or equivalently the duality gap, for the full problem Eq. ([5]). 
More precisely, the duality gap of the optimization problem Eq. Q is 



Lj{wj) + L*ji-Kj) + ^ [njiwj)f + ^ miKj)]' 
[Ljiwj) + L*j{-Kj) + u;Jacj} + [^jiwj)]" + ^ 



which is a sum of two nonnegative terms, the nonnegativity comin g from the Fenchel-Young in- 
equality ( Borwein and Lewis , 20061 : Boyd and Vandenberghe . 2004, Proposition 3.3.4 and Section 
3.3.2 respectively). We can think of this duality gap as the sum of two duality gaps, respectively 
relative to Lj and Oj. Thus, if we have a primal candidate wj and we choose kj = —VLj{wj), 
the duality gap relative to Lj vanishes and the total duality gap then reduces to 



In order to check that the reduced solution wj is optimal for the full problem in Eq. we pad 
wj with zeros on J'^ to define w and compute k = —VL{w), which is such that kj = —VLj{wj). 
For our given candidate pair of primal/dual variables {w,k}, we then get a duality gap for the full 
problem in Eq. ^ equal to 



Computing this gap requires computing the dual norm which itself is as hard as the original problem, 
prompting the need for upper and lower bounds on Q* (see Propositions |4] and [5] for more details). 
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Figure 7: The active set (black) and the candidate patterns of variables, i.e. the variables in K\J 
(hatched in black) that can become active. The fringe groups ai^e exactly the groups that 
have the hatched areas (i.e., here we have J'j = [jKeu-p{J) ^k\Qj = {Gi, G2, G3}). 




Figure 8: The active set (black) and the candidate patterns of variables, i.e. the variables in 
K\J (hatched in black) that can become active. The groups in red are those in 
^KdYiviJ) ^k\Qj, while the blue group is in -7v\(U/<6np(j) Gk\Qj)- The blue group 
does not intersect with any patterns in Hp (J). 
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4.2 Active set algorithm 

We can interpret the active set algorithm as a walk through the DAG of nonzero patterns allowed by 
the norm $7. The parents Hp (J) of J in this DAG are exactly the patterns containing the variables 
that may enter the active set at the next iteration of Algorithm [3] The groups that are exactly at 
the boundaries of the active set (referred to as the fringe groups) are Tj = {G G {GjY ', $G' e 
{QjY, G C G'}, i.e., the groups that are not contained by any other inactive groups. 

In simple settings, e.g., when Q is the set of rectangular groups, the correspondence between 
groups and variables is straightforward since we have Tj = \^j^^yi-p{j)Sk\Gj (see Figure |71l. 
However, in general, we just have the inclusion (UA'enp(j) Sk\Gj) C Tj and some elements of 
Tj might not correspond to any patterns of variables in n-p( J) (see Figure[8l). 

We now present the optimality conditions (see proofs in Appendix 10) that monitor the progress 
of Algorithm [3 

Proposition 4 (Necessary condition) Ifw is optimal for the full problem in Eq. dJ]), then 

\\VL{w)k\j\\. 



max 



i^enp(j) EHefe\Gj II 

'^if\j| 

Proposition 5 (Sufficient condition) If 



<{-\w^VL{w)Yk (N) 



then w is an approximate solution for Eq. Q whose duality gap is less than e > 0. 

Note that for the Lasso, the conditions {N) and (^o) (i.e., the sufficient condition taken with 
e = 0) are both equivalent (up to the sq uaring of f^) to the condition \\yL{w) .]c \\^ < —w^VL{w), 
which is the usual optimality condition ( Wainwright , 20091 : Tibshirani . 1996). Moreover, when they 



are not satisfied, our two conditions provide good heuristics for choosing which K G Hp (J) should 
enter the active set. 

More precisely, since the necessary condition ((TVT) directly deals with the variables (as opposed 
to groups) that can become active at the next step of Algorithm |3l it suffices to choose the pat- 
tern K G n-p(J) that violates most the condition. 

The heuristics for the sufficient condition ( [5^ implies to go from groups to variables. We simply 
consider the group G G Fj that violates most the sufficient condition and then take all the patterns 
of variables K G np( J) such that n G / to enter the active set. If G n (Uft:enp(j) ^) = '^J, 
we look at all the groups H G Fj such that H n G ^ and apply the scheme described before 
(see Algorithmic. 

A direct consequence of this heuristics is that it is possible for the algorithm to jump over the 
right active set and to consider instead a (slightly) larger active set as optimal. However if the active 
set is larger than the optimal set, then (it can be proved that) the sufficient condition (^o) is satisfied, 
and the reduced problem, which we solve exactly, will still output the correct nonzero pattern. 

Moreover, it is worthwhile to notice that in Algorithm [3l the active set may sometimes be in- 
creased only to make sure that the current solution is optimal (we only check a sufficient condition 
of optimality). 
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Algorithm 3 Active set algorithm 

Input: Data {{xi,yi), i = 1, . . . ,n}, regulaiization parameter A, 
Duality gap precision e, maximum number of variables s. 
Output: Active set J, loading vector w. 
Initialization: J = {0}, w = 0. 
while ( (A^) is not satisfied ) and ( | J| < s ) do 
Replace J by violating K G n-p( J) in (Nl . 

Solve the reduced problem min^^gj^iJi L j{wj) + ^ [Vtj{wj)f' to get w. 
end while 

while ( {Sg) is not satisfied ) and ( | J| < s ) do 
Update J according to Algorithmic 

Solve the reduced problem min^^gjj|j| L j{wj) + ^ [Vtj{wj)f' to get w. 
end while 



Convergence of the active set algorithm. The procedure described in Algorithm[3]can terminate 
in two different states. If the procedure stops because of the limit on the number of active variables 
s, the solution might be suboptimal. Note that, in any case, we have at our disposal a upperbound 
on the duality gap. 

Otherwise, the procedure always converges to an optimal solution, either (1) by validating both 
the necessary and sufficient conditions (see Propositions |4] and [5]), ending up with fewer than p 
active variables and a precision of (at least) e, or (2) by running until the p variables become active, 
the precision of the solution being given by the underlying solver. 



Algorithm 4 Heuristics for the sufficient condition 



Let G G Fj be the group that violates (j^J) most, 
if (G'n(Ui,enp(j)A')/0) then 

for K G Hp (J) such that K n G / do 
J ^ JlJK. 

end for 
else 

for H such that n G / do 
for K G np( J) such that n ii^ / do 

J ^ JUK. 
end for 
end for 
end if 



Algorithmic complexity. We analyze in detail the time complexity of the active set algorithm 
when we consider sets of groups Q such as those presented in the examples of Section 13.51 We 
recall that we denote by Q the set of orientations in Q (for more details, see Section 1331 ). 

For such choices of Q, the fringe groups Fj reduces to the largest groups of each orientation 
and therefore < |0|. We further assume that the groups in Qq are sorted by cardinaUty, so that 
computing Tj costs O(|0|). 
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Given an active set J, both the necessary and sufficient conditions require to have access to the 
direct parents Ilp{J) of J in the DAG of nonzero patterns. In simple settings, e.g., when Q is the 
set of rectangular groups, this operation can be performed in 0(1) (it just corresponds to scan the 
(up to) four patterns at the edges of the cun^ent rectangular hull). 

However, for more general orientations, computing Il-p{J) requires to find the smallest nonzero 
patterns that we can generate from the groups in J^j, reduced to the stripe of variables around the 
cuiTcnt hull. This stripe of variables can be computed as [{Jceig j^XTj ^] \'^^ ^^^^ getting 
np(J) costs 0(s2l®l +p\g\) in total. 

Thus, if the number of active variables is upper bounded by s<^p (which is true if our target is 
actually sparse), the time complexity of Algorithm |3] is the sum of: 

• the computation of the gradient, 0{snp) for the square loss. 

• if the underlying solver called upon by the active set algorithm is a standard SOCP solver, 
0(s maxjgp I j|<g l^jp'^ + s^'^) (note that the term s^-^ could be improved upon by using 
warm-restait strategies for the sequence of reduced problems). 

• ti times the computation of (iV), that is 0(ti(s2l®l + p\g\ + sn^) + p\Q\) = 0{tip\Q\). 

During the initialization (i.e., J = 0), we have |n-p(0)| = 0{p) (since we can start with any 
singletons), and = \Gk\ = which leads to a complexity of 0(p|^|) for the sum 

YlGegK\G J ~ X^Gefe' ^^^^ however that this sum does not depend on J and can therefore 
be cached if we need to make several runs with the same set of groups Q. 

• t2 times thecomputation of (S'e),thatisO(t2(s2l®l+p|a| + |6p + |G|p+p|^|)) = 0{t2p\G\), 
with ti + t2 < s. 

We finally get complexity with a leading term in 0(sp|t/| + s maxjg-p | j|<5 l^jp'^ + s^'^), 
which is much better than 0{p^'^ + |^|'^'^), without an active set method. In the example of the 
two-dimensional grid (see Section 1331 ). we have \Q\ = 0{^) and 0(s max{p^'^^, s^'^}) as total 
complexity. The simulations of Section [6] confirm that the active set strategy is indeed useful when 
s is much smaller than p. Moreover, the two extreme cases where s p or p ^ 1 are also shown 
not to be advantageous for the active set strategy, since either it is cheaper to use the SOCP solver 
directly on the p variables, or we uselessly pay the additional fixed-cost of the active set machinery 
(such as computing the optimality conditions). Note that we have derived here the theoretical com- 
plexity of the active set algorithm when we use a SOCP method as underlying solver. With the first 
order method presented in Appendix iHl we would instead get a total complexity in 0{sp^-^). 



4.3 Intersecting Nonzero Patterns 

We have seen so far how overlapping groups can encore prior information about a desired set of 
(non)zero patterns. In practice, controlling these overlaps may be delicate and hinges on the choice 
of the weights {dP)Geg (see the experiments in Section |6l). In particular, the weights have to take 
into account that some variables belonging to several overlapping groups are penalized multiple 
times. 

However, it is possible to keep the benefit of overla pping gr oups whilst limiting their side effects , 
by taking up the idea of support intersection ( Bach . boOSai : iMeinshausen and Buhlmann . 2008 ). 
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First introduced to stabilize the set of variables recovered by the Lasso, we reuse this technique in a 
different context, based on the fact that Z is closed under union. 

If we deal with the same sets of groups as those considered in Section 13.51 it is natural to 
rewrite Q as Ueee^^' where is the set of the orientations of the groups in Q (for more details, see 
Section [331 ). Let us denote by w and the solutions of Eq. where the regularization term is 
respectively defined by the groups in Q and by the group^ in Qq. 

The main point is that, since V is closed under intersection, the two procedures described below 
actually lead to the same set of allowed nonzero patterns: 

a) Simply considering the nonzero pattern of w. 

b) Taking the intersection of the nonzero patterns obtained for each , in Q. 

With the latter procedure, although the learning of several models is required (a number of 
times equals to the number of orientations considered, e.g., 2 for the sequence, 4 for the rectangular 
groups and more generally |©| times), each of those learnings involves a smaller number of groups 
(that is, just the ones belonging to Qq). In addition, this procedure is a variable selection technique 
that therefore needs a second step for estimating the loadings (restricted to the selected nonzero 
pattern). In the experiments, we follow iBachI (l2008ah and we use an ordinary least squares (OLS). 



The simulations of Section |6] will show the benefits of this variable selection approach. 
5. Pattern Consistency 

In this section, we analyze the model consistency of the solution of Eq. Q for the square loss. 
Considering the set of nonzero patterns V derived in Section |3l we can only hope to estimate the 
correct hull of the generating sparsity pattern, since Theorem |2] states that other patterns occur with 
zero probability. We derive necessary and sufficient conditions for model consistency in a low- 
dimensional setting, and then consider a high-dimensional result. 

We consider the square loss and a fixed-design analysis (i.e., xi, . . . , 2;„ are fixed). The exten- 
sion of the follo wing consiste ncy results to other loss functions is beyond the scope of the paper 
(see for instance Bach . 2009 ah . We assume that for alH G {1, . . . , n}, yi = w^Xj + where the 



vector e is an i.i.d. vector with Gaussian distributions with mean zero and variance o"^ > 0, and 
w G is the population sparse vector; we denote by J the ^-adapted hull of its nonzero pattern. 
Note that estimating the ^-adapted hull of w is equivalent to estimating the nonzero pattern of if 
and only if this nonzero pattern belongs to V. This happens when our prior information has led us 
to consider an appropriate set of groups Q. Conversely, if Q is misspecified, recovering the hull of 
the nonzero pattern of w may be irrelevant, which is for instance the case if w = (^^) € and 
Q = {{1},{1,2}}. Finding the appropriate structure of Q directly from the data would therefore be 
interesting future work. 

5.1 Consistency Condition 

We begin with the low-dimensional setting where n is tending to infinity with p fixed. In addition, 
we also assume that the design is fixed and that the Gram matrix Q = ^ Yl'i=i ^i^J invertible 



6. To be more precise, in order to regularize every variable, we add the full group {1, . . . ,p} to Qe, which does not 
modify V. 
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with positive-definite (i.e., invertible) limit 

lim Q = Q ^ 0. 

In this setting, the noise is the only source of randomness. We denote by rj the vector defined as 



■Gegj,G5j 

In the Lasso and group Lasso setting, the vector rj is respectively the sign vector sign(v^j) and the 
vector defined by the blocks (||^^|-)Geej- 

We define i}j{wjc) = J2G&{gjY W^j'^ ° ^J'^ II2 (which is the norm composed of inactive groups) 
with its dual norm {^j)*; note the difference with the norm reduced to J^, defined as i^jc[wjc) = 
Eceg \\djcOWjc\\^. 

The following Theorem gives the sufficient and necessary conditions under which the hull of 
the genera ting pattern i s con sistently estimated. Those conditions naturally extend the results of 
Zhao and Yu (>200d) and lBach (i2008bi ) for the Lasso and the group Lasso respectively (see proof in 



Appendix 10). 

Theorem 6 (Consistency condition) Assume // — 0, f^^/n — 00 in Eq. ([2]). If the hull is consis- 
tently estimated, then (^^j)*[Qj<=jQjjrj] < 1- Conversely, /J (r2j)*[QjcjQjjrj] < 1, then the 
hull is consistently estimated, i.e., 

F{{j e{l,...,p},wj^0} = 3) 1. 

n— >+oo 

The two previous propositions bring into play the dual norm (fij)* that we cannot compute 
in closed form, but requires to solve an optimization problem as complex as the initial problem in 
Eq. ([5]). However, we can prove bounds similar- to those obtained in Propositions |4] and |5] for the 
necessary and sufficient conditions. 

Comparison with the Lasso and group Lasso. For the £i-norm, our two bounds lead to the 
usual consistency conditions for the Lasso, i.e., the quantity ||QjcjQjjSign(w^j)||oo must be less 
or strictly less than one. Similarly, when Q defines a partition of {1, ... ,p} and if all the weights 
equal one, our two bounds lead in turn to the consistency conditions for the group Lasso, i.e., 

the quantity maxGe(gj)c ||Qg Hu11(J)Qhuii(j)hu11(j) ( ||wq||^ )Gggj lb must be less or strictly less than 
one. 



5.2 High-Dimensional Analysis 



We prove a high-dimensional variab le consistency result (see proof in Appen dix iGb that extends the 
corresponding result for the Lasso ( Zhao and Yu . 20061 : Wainwright . 2009h . by assuming that the 
consistency condition in Theorem |6]is satisfied. 

Theorem 7 Assume that Q has unit diagonal, k = Amin(Qjj) > and (^^j)*[Qjcj<5jjrj] < 
1 — T, with T > 0. If Tfi^/n > aC3{G,J), and /u|J|^/^ < C4{Q,J), then the probability of 
incorrect hull selection is upper bounded by: 



exp 



n//VCi(g,J) 
2^2 



+ 2| J| exp 



nC2(g,J) 
2|J|a2 
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where Ci{Q,J), C2(^, J), C-i{Q,3) and C4(^, J) are constants defined in Appendix [G] which 
essentially depend on the groups, the smallest nonzero coefficient of w and how close the support 
{j G J : Wj 7^ 0} o/w is to its hull J, that is the relevance of the prior information encoded by Q. 

In the Lasso case, we have Ci (a, J) = 0{l),C2{Q,3) = 0{\3\~'^),Cz{g,3) = 0{{\ogpY/^) 
and C4(^, J) = 0(1 J|~^), leading to the usual scaling n \ogp and ^ (T(logp/n)^/^. 

We can also give the scaling of these constants in simple settings where groups overlap. For 
instance, let us consider that the variables are organized in a sequence (see Figure SJ). Let us further 
assume that the weights {d^)Geg satisfy the following two properties: 

a) The weights take into account the overlaps, that is, 

d'f = ^{\{H e g ; H 3 j, H C G and H ^ G}\), 
with 1 1— /3(t) G (0, 1] a non-increasing function such that /3(0) = 1, 

b) The term 



is upper bounded by a constant /C independent of p. 

Note that we consider such weights in the experiments (see Section|6ll. Based on these assumptions, 
some algebra directly leads to 



We thus obtain a scaling similar to the Lasso (with, in addition, a control of the allowed nonzero 
patterns). 

With stronger assumptions on the possible positions of J, we may have better scalings, but these 
are problem-dependent (a careful analysis of the group-dependent constants would still be needed 
in all cases). 

6. Experiments 

In this section, we carry out several experiments to illustrate the behavior of the sparsity-inducing 
norm Q,. We denote by Structured-lasso, or simply Slasso, the models regularized by the norm Q. In 
addition, the procedure (introduced in Section 1431 ) that consists in intersecting the nonzero patterns 
obtained for different models of Slasso will be referred to as Intersected Structured-lasso, or simply 
ISlasso. 

Throughout the experiments, we consider noisy linear models Y = w^X + e, where w G 
is the generating loading vector and e is a centered Gaussian noise with its variance set to satisfy 
||w^X||2 = 3 ||e||2- This consequently leads to a fixed signal-to-noise ratio. We assume that the 
vector w is sparse, i.e., it has only a few nonzero components, that is, | J| <^ p. We further assume 
that these nonzero components are either organized on a sequence or on a two-dimensional grid (see 
Figure |9ll. Moreover, we consider sets of groups Q such as those presented in Section [331 We also 
consider different choices for the weights {d^)Geg that we denote by (Wl), (W2) and (W3) (we 
will keep this notation in the following experiments): 




will < ^{u) < 2/C \\u 



1 ' 



for all ueRP. 
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(Wl): uniform weights, df^ = 1, 

(W2): weights depending on the size of the groups, = |G|~^, 

(W3): weights that take into account overlapping groups, = p\i^^^ '^^i' HcG and h^g}\^ 
some p G (0, 1). 

For each orientation in Q, the third type of weights (W3) aims at reducing the unbalance caused by 
the overlapping groups. Specifically, given a group G € Q and a variable j G G, the corresponding 
weight dj is all the more small as the variable j already belongs to other groups with the same 
orientation. 

Unless otherwise specified, we use the third type of weights (W3) with p = 0.5. In the following 
experiments, the loadings wj, as well as the design matrices, are generated from a standard Gaussian 
distribution with identity covariance matrix. The positions of J are also random and are uniformly 
drawn. 

Prediction error and relevance of the structured prior. We show in this experiment that the 
prior information we put through the norm Q improves upon the predictive power. We are looking 
at two situations where we can express a structural prior through Q, namely (1) the selection of a 
contiguous pattern on a sequence and (2) the selection of a convex pattern on a grid (see Figure |9l). 

In what follows, we consider p = 400 vaiiables with generating patterns w whose hulls have a 
constant size of | J| = 24 variables. In order to evaluate the relevance of the contiguous (or convex) 
prior, we also vary the number of zero variables that are contained in the hull (see Figure |9ll. We 
then compute the prediction error for different sample sizes n G {250, 500, 1000}. The prediction 
enor is understood here as 

\\X'''\^-w)\\l 



where w denotes the estimate of the OLS, performed on the nonzero pattern found by the model 
considered (i.e., either Lasso, Slasso or ISlassofl The regularization parameter is chosen by 5- 
fold cross-validation and the test set consists of 500 samples. For each value of n, we display on 
Figure [TT] and Figure [T2]the median errors over 50 random settings {J, w, X, e}, for respectively 
the sequence and the grid. 

First and foremost, the simulations highlight how important the weights {d^)Geg P^" 
ticular, the uniform (Wl) and size-dependent weights (W2) perform poorly since they do not take 
into account the overlapping groups. The models learned with such weights do not manage to re- 
cover the correct nonzero patterns (and even worse, they tend to select every variable — see the right 
column of Figure [TTI). 

Although groups that moderately overlap do help (e.g., see the Slasso with the weights (W3) on 
the left column of Figure [TTI). it remains delicate to handle groups with many overlaps, even with an 
appropriate choice of {d°)G£g (e.g., see the right column of Figure [T2l where Slasso considers up 
to 8 overlaps on the grid). The ISlasso procedure does not suffer from this issue since it reduces the 
number of overlaps whilst keeping the desirable effects of overlapping groups. Another way to yield 
a better leve l of sparsity, even wit h many overlaps, would be to consider non-convex alternatives to 
(see, e.g., Jenatton et al. . 2010|) . Naturally, the benefit of ISlasso is more significant on the grid 



7. Even though we make comparisons based on prediction errors, the experiments illustrate the results from Section[5] 
since we first use our method as a variable selection step. 
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Hull with 33% of nonzero variables 



Hull with 25% of nonzero variables 




Hull with 50% of nonzero variables 



Hull with 50% of nonzero variables 



Hull with 83% of nonzero variables 



Hull with 75% of nonzero variables 



4 



Hull with 1 00% of nonzero variables 



Hull with 100% of nonzero variables 



Figure 9: Examples of generating patterns (the zero variables are represented in black, while the 
nonzero ones are in white): (Left column, in white) generating patterns that are used for 
the experiments on 400-dimensional sequences; those patterns all form the same hull of 
24 variables (i.e., the contiguous sequence in the bottom left figure). (Right column, in 
white) generating patterns that we use for the 20x20-dimensional grid experiments; again, 
those patterns all form the same hull of 24 variables (i.e., the diamond-shaped convex in 
the bottom right figure). The positions of these generating patterns are randomly selected 
during the experiments. For the grid setting, the hull is defined based on the set of groups 
that are half-planes, with orientations that are multiple of 7r/4 (see Section [33]) . 
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than on the sequence as the latter deals with fewer overlaps. Moreover, adding the ib7r/4-groups to 
the rectangular groups enables to recover a nonzero pattern closer to the generating pattern. This is 
illustrated on the left column of Figure [12] where the eiTor of ISlasso with only rectangular- groups 
(in black) corresponds to the selection of the smallest rectangular box around the generating pattern. 

On the other hand, and more importantly, the experiments show that if the prior about the gen- 
erating pattern is relevant, then our structured approach performs better that the standard Lasso. 
Indeed, as displayed on the left columns of Figure [TT] and Figure [121 as soon as the hull of the 
generating pattern does not contain too many zero variables, Slasso/ISlasso outperform Lasso. In 
fact, the sample complexity of the Lasso depends on the number of nonzero variables in w as op- 
posed to the size of the hull for Slasso/ISlasso. This also explains why the error for Slasso/ISlasso 
is almost constant with respect to the number of nonzero variables (since the hull has a constant 
size). Note finally that, even though our structured approach does not always dramatically outper- 
form the standard unstructured approach in terms of prediction, it has the advantage of being more 
interpretable. 
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■A-SOCP (7C/4) 
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log(Number of variables) 
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Figure 10: Computational benefit of the active set algorithm: CPU time (in seconds) versus the 
number of variables p, displayed in log-log scale. The points, the lower and upper en^or 
bars on the curves respectively represent the median, the first and third quartile. Two 
sets of groups Q are considered, the rectangular groups with or without the ibvr / 4-groups 
(denoted by (vr/4) in the legend). Due to the computational burden, we could not obtain 
the SOCP's results for p = 2500. 
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Sample size n=500 



Sample size n=500 
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25 50 75 100 

Proportion of nonzero variables in the hull (%) 



,-3 Sample size n=1 000 Sample size n=1 000 




Proportion of nonzero variables in the hull (%) Proportion of nonzero variables in the hull (%) 



Figure 11: Experiments on the sequence: for the sample sizes n G {250, 500, 1000}, we plot the 
prediction enor versus the proportion of nonzero variables in the hull of the generating 
pattern. We display the results on two different columns since the models obtain very 
heterogeneous performances (on the right column, the error is in log scale). The points, 
the lower and upper error bars on the curves respectively represent the median, the first 
and third quartile, based on 50 random settings {J, w, X, e}. 
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Figure 12: Experiments on the grid: for the sample sizes n G {250, 500, 1000}, we plot the pre- 
diction error versus the proportion of nonzero variables in the hull of the generating 
pattern. We display the results on two different columns since the models obtain very 
heterogeneous performances (on the right column, the error is in log scale). The points, 
the lower and upper error bars on the curves respectively represent the median, the first 
and third quartile, based on 50 random settings {3,-w,X,e}. Two sets of groups Q 
are considered, the rectangular groups with or without the ib7r/4-groups (denoted by 
(7r/4) in the legend). In addition, we have dropped for clarity the models that performed 
already poorly on the sequence. 
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Active set algorithm. We finally focus on the active set algorithm (see Section Ull and com- 
pare its time complexity to the SOCP solver when we are looking for a sparse structured tar- 
get. More precisely, for a fixed level of sparsity |J| = 24 and a fixed number of observations 
n = 3500, we analyze the complexity with respect to the number of variables p that varies in 
{100, 225, 400, 900, 1600, 2500}. 

We consider the same experimental protocol as above except that we display the median CPU 
time based onljj^on 5 random settings {J, w, X, e}. 

We assume that we have a rough idea of the level of sparsity of the true vector and we set the 
stopping criterion s = 4| J| (see Algorithm |3]), which is a rather conservative choice. We show on 
Figure [lO] that we considerably lower the computational cost for the same level of performanc^ 
As predicted by the complexity analysis of the active set algorithm (see the end of Section lU, 
considering the set of rectangular groups with or without the ib7r/4-groups results in the same 
complexity (up to constant terms). We empirically obtain an average complexity of w 0(p^-^^) for 
the SOCP solver and of 0{p^'^^) for the active set algorithm. 

Not surprisingly, for small values of p, the SOCP solver is faster than the active set algorithm, 
since the latter has to check its optimality by computing necessary and sufficient conditions (see 
Algorithm |3] and the discussion in the algorithmic complexity paragraph of Section|4|. 



7. Conclusion 



We have shown how to incorporate prior knowledge on the form of nonzero patterns for linear 
supervised learning. Our solution reUes on a regularizing term which linearly combines ^2-norms 
of possibly overlapping groups of variables. Our framework brings into play intersection-closed 
families of nonzero patterns, such as all rectangles on a two-dimensional grid. We have studied 
the design of these groups, efficient algorithms and theoretical guarantees of the structured sparsity- 
inducing method. Our experiments have shown to which extent our model leads to better prediction, 
depending on the relevance of the prior information. 

A natural extension to this work is to conside r bootstrapping since this may improve theoretica l 
guarantees and result in better variable selection (lBachLl2008al : lMeinshausen and Buhlmannl. 120081) . 
In order to deal with broader families of (non)zero patterns, it would be interes t ing to c ombine our 



approa ch with recent work on structured sparsity: for instance. IB araniuk et al I (I20()8r ): iJacob et al. 
(2009) consider union-closed collecti ons of nonze r o patt erns. He and CarinI ( 20091) exploit struc- 



ture through a Bayesian prior while Huang et al. ( 2009|) handle non-convex penalties based on 
information-theoretic criteria. 

More generally, our regularization scheme could also be used for various learning tasks, as soon 
as prior k nowledge on the structure of the sparse representat ion is available, e.g., for multiple kernel 



learning ( Micchelli and Pontii 2006h . multi-task learning ( Argvriou et al. . 2008 : l Obozinski et al 
2009 : Kim and Xingl.l2009l) and sparse matrix factorization problems ( Mairal et al. , 201C)[ 
2010) . 



Jenatton et al. 



Finally, although we have mostly explored in this paper the algorithmic and theoretical issues 
related to these norms, this type of prior knowledge is of clear interest for the spatially and tempo- 



8. Note that it already corresponds to several hundreds of runs for both the SOCP and the active set algorithms since we 
compute a 5-fold cross-validation for each regularization parameter of the (approximate) regularization path. 

9. We have not displayed this second figure that is just the superposition of the error curves for the SOCP and the active 
set algorithms. 
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rally structured data typic al in bioinformatics, computer vision and neuroscience applications (see, 
e.g., 



Jenatton etallboioh . 
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Appendix A. Proof of Proposition [T] 

We recall that L{w) = ^ J2^=i ^{yi^w~^ ^i) ■ Since w Q{w) is convex and goes to infinite when 
ll^llg goes to infinite, and since L is lower bounded, by Weierstrass' theorem, the problem in Eq. Q 
admits at least one global solution. 
•First case: Q invertible. The Hessian of L is 



1 " ^ 
1=1 ^ 



It is positive definite since Q is positive definite and minjgji ^} ■^^{yi,w'^ Xi) > 0. So L is 
strictly convex. Consequently the objective function L + ^uJl is strictly convex, hence the uniqueness 
of its minimizer. 

•Second case: {1, . . . ,p} belongs to Q. We prove the uniqueness by contradiction. Assume that 
the problem in Eq. Q admits two different solutions w and w. Then one of the two solutions is 
different from 0, say u; / 0. 

By convexity, it means that any point of the segment [w, w] = \^aw + (1 — a)w; a G [0, 1]} 
also minimizes the objective function L + ^Q.. Since both L and i^lQ are convex functions, it means 
that they are both linear when restricted to [u;, 

Now, /xfi is only linear on segments of the form [v, tv] with v and t > 0. So we necessarily 
have w = tw for some positive t. We now show that L is strictly convex on [w,tw], which will 
contradict that it is linear on [w, w]. Let E = Span(xi, . . . , Xn) and i?-*" be the orthogonal of E 
in W. The vector w can be decomposed in w = w' + w" with w' ^ E and w" G E-^. Note that 
we have w' (since if it was equal to 0, w" would be the minimizer of ^$7, which would imply 
w" = and contradict w / 0). We thus have {w~^xi, . . . , w~^Xn) = {w'~^xi, . . . , w'~^ Xn) / 0. 

This implies that the function s i— )• £{yi, sw~^Xi) is a polynomial of degree 2. So it is not linear. 
This contradicts the existence of two different solutions, and concludes the proof of uniqueness. 

Remark 8 Still by using that a sum of convex Junctions is constant on a segment if and only if the 
functions are linear on this segment, the proof can be extended in order to replace the alternative 
assumption "{ 1 , . . . , p} belongs to Q " by the weaker but more involved assumption: for any (j, k) £ 
{1, . . . there exists a group G £ Q which contains both j and k. 

Appendix B. Proof of Theorem |2] 

For w G W, we denote by Z{w) its zero pattern (i.e., the indices of zero-components of w). To 
prove the result, it suffices to prove that for any set / C {I, . . . ,p} with I'^ ^ Z and |/| < — 1, 
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the probability of 

Sj = {Y £ M": there exists w solution of the problem in Eq. © with Z{w) = I'^] 

is equal to 0. We will prove this by contradiction: assume that there exists a set / C {1, . . . ,p} with 
I" Z,\I\<k-l£Lnd F{£j) > 0. Since F ^ Z, there exists a G Hull(/) \ /. Let J = / U {a} 
and ^7 = {G G C/ : Gn/ / 0} be the set of active groups. Define R-^ = {w eW : wjc = 0}. The 
restrictions Lj : M*^ — )• M and ilj : M'^ — )• R of L and are continuously differentiable functions 
on |u; G M-^ : iM/ / 0} with respective gradients 



VLj{w) = (I^h) ^ and Vnj{w) = lwj(^ ^ {dff\\d'^ 



o w 




G3j 

Let/('u;,y) = VLj(?i;)+/iVr2j(tt;), where the dependence in y of /(lu, y) is hidden in VLj('u;) = 

For Y G £i, there exists w G M-^ with Z{w) = J'^, which minimizes the convex function 
Lj + //Oj. The vector w satisfies f{w, Y) = 0. So we have proved Ej C 8'^, where 

S'j = {Y £W : there exists G R-^ with ^(w) = r and /(w, y) = O). 

Let y G Consider the equation f{w,y) = on G M'' : wj / for any j G /}. By 

construction, we have \J\ < k, and thus, by assumption, the matrix X"^ = {xn)j)~^ G 

]^nx|j| j^^g j-^jj, I j| jj^g proof of Proposition [TJ this implies that the function Lj is strictly 

convex, and then, the uniqueness of the minimizer of Lj + fi^l, and also the uniqueness of the point 
at which the gradient of this function vanishes. So the equation f{w, y) = on G M"' : wj / 
for any j G /} has a unique solution, which we will write w^. 

On a small enough ball around {w^j, y), / is continuously differentiable since none of the norms 
vanishes at w^j. Let {fj)jej be the components of / and Hjj = {§ij^) j^j f^^j- The matrix Hjj is 
actually the sum of: 



a) the Hessian of Lj, which is positive definite (still from the same argument as in the proof of 
Theorem [T]), 

b) the Hessian of the norm $7j around {w^j,y) that is po sitive semidefinite on this small ball 
according to the Hessian characterization of convexity (IBorwein and Lewisl. |2006|. Theorem 
3.1.11). 

Consequently, Hjj is invertible. We can now apply the implicit function theorem to obtain that for 
y in a neighborhood of y, 

= v^(y), 

with -0 = a continuously differentiable function satisfying the matricial relation 

(...,V^,-,...)i/jj + (...,Vy/„...) = 0. 

Let Ca denote the column vector of i/jj corresponding to the index a, and let D the diagonal 
matrix whose {i, i)-th element is g^g^/ {Vi , w'^Xi). Since n(. . . , Vyfj, ■ ■ ■) = DX^ , we have 

nVipa = -DX-^Ca. 
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Now, since X has full rank, Cq, / and none of the diagonal elements of D is null (by assumption 
on t), we have Vif^a 7^ 0. Without loss of generality, we may assume that dipa/dyi 7^ on a 
neighborhood of y. 

We can apply again the implicit function theorem to show that on an open ball in M" centered 
at y, say By, the solution to ipa^X) = can be written yi = ip{y2, • • • , y-n) with ip a continuously 
differentiable function. 

By Fubini's theorem and by using the fact that the Lebesgue measure of a singleton in equals 
zero, we get that the set A{y) = {Y ^ By : ipaiY) = O} has thus zero probability. Let 5 C be 
a compact set. We thus have S <Z S'j. 

By compacity, the set S can be covered by a finite number of ball By. So there exist yi, . . . ,ym 
such that we have S C A{yi) U • • • U A{ym)- Consequently, we have P(5) = 0. 

Since this holds for any compact set in £j and since the Lebesgue measure is regular, we have 
P{£i) = 0, which contradicts the definition of /, and concludes the proof. 

Appendix C. Proof of the minimality of the Backward procedure (see Algorithm [B 

There are essentially two points to show: 

• Q spans Z. 

• ^ is minimal. 

The first point can be shown by a proof by recurrence on the depth of the DAG. At step t, the base 
verifies {UggS' ^' - ^^^^^ = {G e Z, \G\ < t} because an element G € Z is either the 

union of itself or the union of elements strictly smaller. The initialization t = mmcez \ G\ is easily 

verified, the leafs of the DAG being necessarily in Q. 

As for the second point, we proceed by contradiction. If there exists another base Q* that spans 

Z such that Q* C G, then 

By definition of the set Z, there exists in turn Q' <^ Q*, G' ^ {e} (otherwise, e would belong to 
Q*), verifying e = UceS' which is impossible by construction of Q whose members cannot be 
the union of elements of Z. 



Appendix D. Proof of Proposition |3] 



The proposition comes from a classic result of Fenchel Duality (|Borwein and LewisLl2006l. Theorem 
3.3.5 and Exercise 3.3.9) when we consider the convex function 



A 2 
hj :wj^ - [nj{wj)] , 



whose Fenchel conjugate h*j is given by kj 1-^ ^ (IBoyd and Vandenberghd . 12004, ex- 

ample 3.27). Since the set 



{wj £ M'"^!; hj{'wj) < 00} n {wj G M'"^'; Lj{wj) < 00 and Lj is continuous at wj} 
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is not empty, we get the first part of tlie proposition. Moreover, the primal-dual variables {wj, kj} 
is optimal if and only if 

\-Kj G dLj{wj), 

[kj G d[^[Qj{wj)f]=XQj{wj)dQj{wj), 

where d^j{wj) denotes the subdifferential of 17 j at wj. The differentiability of Lj at w j then 
gives dLj{wj) = {VLj{wj)}. It now remains to show that 

KJ G \Vlj{wj)dVlj{wj) (8) 

is equivalent to 

w]Kj = ^[^rj{Kj)f = X[nj{wj)f . (9) 



As a starting point, the Fenchel-Young inequality ( Borwein and Lewis , 20061 Proposition 3.3.4) 
gives the equivalence between Eq. ^ and 

^ [^j{wj)f + ^ [mi^j)f = w']kj. (10) 



In addition, we have (|Rockafellan . Il970h 



dnj{wj) = {uj G rI-^I; nJu^J = ^j{wj) and ^*j{uj) < 1}. (11) 
Thus, if Kj G XQj{'Wj)dQj{wj) then w~jKj = A ['nj{wj)]'^ . Combined with Eq. (fTOl) . we obtain 

w]Kj = {{n*j{Kj)f. 

Reciprocally, starting from Eq. we notably have 

wjKj = A [nj{wj)f . 

In light of Eq. (ITTI ). it suffices to check that < XQ,j{wj) in order to have Eq. ([8]l. Combining 

Eq. ^ with the definition of the dual norm, it comes 

i [n}{Kj)f = w]kj < n*j{Kj)nj{wj), 

which concludes the proof of the equivalence between Eq. ^ and Eq. 
Appendix E. Proofs of Propositions IH and |5] 

In order to check that the reduced solution wj is optimal for the full problem in Eq. we complete 
with zeros on J'^ to define w, compute k = —VL{w), which is such that kj = —VLj{wj), and 
get a duality gap for the full problem equal to 



^ ([^*{K)f - Xw-jKj 



2A 

By designing upper and lower bounds for Q,*{k), we get sufficient and necessary conditions. 
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E.l Proof of Proposition |4] 

Let us suppose that w* = is optimal for the full problem in Eq. Following the same 

derivation as in Lemma[T4](up to the squaring of the regularization O), we have that w* is a solution 
of Eq. © if and only if for all u G W, 

u^VL{w*) + \VL{w*){u]rj + {Vfj)[ujc]) > 0, 

with 

_ X - dp od° o w* 

We project the optimality condition onto the variables that can possibly enter the active set, i.e., the 
variables in np( J). Thus, for each K G n-p( J), we have for all uk\j G M'^^^"^', 



ul\ jVL{w*)k\j + X^{w*) ^ d'^\j ° ucnK\.j 



> 0. 

2 



By combining Lemma [T3] and the fact that Qk\j H {QjY = Gk\Gj, we have for all G G Qk\Qj, 
K\J C G and therefore UQf^x\j = '^k\j- Since we cannot compute the dual norm of Ux\j '-^ 
\\^K\j ° '^K\j\\2 in closed-form, we instead use the following upperbound 



dK\j ° ^K\J 
so that we get for all uk\j G ]RI^'\"^I, 



< W 



uIc\.j^L{w*)k\j + \n{w*) ^ l|d^\jl|oo ||ui^\j||2 > 0. 

Finally, Proposition Ogives XQ.{w*) = {— Xw*^ V L{w*)^'^ , which leads to the desired result. 
E.2 Proof of Proposition m 

The goal of the proof is to upper bound the dual norm by taking advantage of the structure 

of ^; we first show how we can upper bound by (J7j)*[kjc]. We indeed have: 

= max k 

< max K 

Y.G&gj\\d-3°^'A\.2-^^G(i{gjr\\'^°°^\\2<^ 

= max K 

= max{0}(Kj),(0})*[A^jc]}, 
where in the last line, we use Lemma [TSl Thus the duality gap is less than 

^ (p*{^f-mKj)f) < ^max{0,P})*[^jc]]2- 
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and a sufficient condition for the duality gap to be smaller than e is 

{n^jY[Kj.]<{2\e+[^:j{Kj)f)h. 

Using Proposition [3l we have —\w^VL{w) = [VL*j{njy^ and we get the right-hand side of Propo- 
sitionlU It now remains to upper bound (J^j)* [kjc]. To this end, we call upon Lemma [TT] to obtain: 



(l^}r[.,.]< max 




Among all groups G € {Gj^, the ones with the maximum values are the largest ones, i.e., those in 
the fringe groups = {G £ {QjY ; ^G" G {GjY-, G C G'}. This argument leads to the result of 
Proposition [5] 

Appendix F. Proof of Theorem |6] 

Necessary condition: We mostly follow the proof of iBachI (|2008bl ') : Izoul (|2006h . Let G M** be the 
unique solution of 

min Liw) + iiVtiw) = min F(w). 

The quantity A = {w — w) / ij, is the minimizer of F defined as 

F(A) = ^A^QA - ^-^g^A + ^"^ [n{w + fiA) - n{w)] , 

where q = ^ Yll=i ^i^i- The random variable //~^g^A is a centered Gaussian with variance 
VATQA/(7i/i2)_ Since Q Q, we obtain that for all A G W, 

/i-ig^A = Op(l). 

Since ^ — )• 0, we also have by taking the directional derivative of at in the direction of A 

[f)(w + ^A) - 0(w)] = rjAj + J^S(AjO + o(l), 

so that for all A G 

F(A) = A^QA + rjAj + 05(Ajc) + Op{l) = Fii^(A) + o.,{l). 
The limiting function F\i^ being stricly convex (because Q ;^ 0) and F bein g convex, we have tha t 



the minimizer A of F tends in probability to the unique minimizer of (IFu and Knighti. 120001) 
referred to as A*. 

By assumption, with probability tending to one, we have J = {j G {1, . . . ,p} ,Wj / 0}, hence 
for any j G 3'^ fiAj = {w — w)j = 0. This implies that the nonrandom vector A* verifies Ajc = 0. 

As a consequence, Aj minimizes Aj QjjAj + rj Aj, hence rj = — QjjAj. Besides, since 
A* is the minimizer of Fum, by taking the directional derivatives as in the proof of Lemma[T4l we 
have 

(J7S)*[Qj.jA}] < L 
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This gives the necessary condition. 



Sufficient condition: We turn to the sufficient condition. We first consider the problem reduced 
to the hull J, 

min Lj{wj) + fii}j{wj). 

that is strongly convex since Qjj is positive definite and thus admits a unique solution wj. With 
similar arguments as the ones used in the necessary condition, we can show that wj tends in proba- 
bility to the true vector wj. We now consider the vector w (^MP which is the vector wj padded with 
zeros on J'^. Since, from Theorem|2l we almost surely have Hull({j G {1, • • • ,p},Wj 7^ 0}) = 
{j S {1, . . . ,p}, Wj 7^ 0}, we have already that the vector w consistently estimates the hull of w 
and we have that w tends in probability to w. From now on, we consider that w is sufficiently close 
to w, so that for any G ^ Qj, \\d'^ o w\\2 ^ 0. We may thus introduce 

Ed'-^ o d'-^ o w 
— — . 

It remains to show that w is indeed optimal for the full problem (that admits a unique solution due 
to the positiveness of Q). By construction, the optimality condition (see Lemma [T4l) relative to the 
active variables J is already verified. More precisely, we have 

VL{w)j + nfj = Qjj{wj - wj) - gj + /ifj = 0. 

Moreover, for all ujc £ M)"^"^, by using the previous expression and the invertibily of Q, we have 

u]cV L{w) jc = u]c {-fJ-QjcjQj^rj + Qj^jQj^qj - Qj'^} ■ 

The terms related to the noise vanish, having actually q = Op(l). Since Q — )■ Q and rj — )• rj, we 
get for all G rI^'I 

u]cVL{w)jc = -/iujc {QjcjQjjrj} + Op(/i). 

Since we assume ($^j)*[Qj^jQjjrj] < 1, we obtain 

-UjcVL{w)jc < /i(Oj)[tijc] + Opifi), 

which proves the optimality condition of Lemma[T4]relative to the inactive variables: w is therefore 
optimal for the full problem. 

Appendix G. Proof of Theorem |7] 

Since our analysis takes place in a finite-dimensional space, all the norms defined on this space are 
equivalent. Therefore, we introduce the equivalence parameters a{3),A{3) > such that 

G rI-^I, a(J) \\u\\^ < Oj[n] < A{J) \\u\\^ . 
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We similarly define a(J'^), ^(J*^) > for the norm (fij) on M^"^"^. In addition, we immediatly get 
by order-reversing: 



Vn G m'-'I, ^(J)^^ \\u\\^ < inj)*[u] < a{J)-^ 



u 



For any matrix F, we also introduce the operator norm ||F||m,s defined as 

\\T\\m,s = sup ||Fn||m- 

|«||s<l 

Moreover, our proof will rely on the control of the expected dual norm for isonormal vectors: 
E [(f^j)*(VF)] with W a centered Gaussian random variable with unit covariance matrix. In the 

case of the La sso, it is of ord er ( log p)^/^. 

Following Bachl ( 2008b ) and Nardi and Rinaldo ( 2008 ). we consider the reduced problem on J, 

mill Lj(t(;j) + 



with solution loj, which can be extended to with zeros. From optimality conditions (see Lemma 
[141 ). we know that 

^*AQ33{w3 - wj) - gj] < ^, (12) 

where the vector q G is defined as q = ^Y17=i^i^i- denote by v = min{|wj|; Wj 7^ 0} 
the smallest nonzero components of w. We first prove that we must have with high probability 
1 1 1 1 00 ^ ^'^^ G € Qj, proving that the hull of the active set of wj is exactly J (i.e., no active 
group is missing). 
We have 

11% - wjII^ < ||Qj]||oo,oo ||<9jj(';i'j - wj)||^ 

< \3\'/\-' {\\Qjj{wj - wj) - qj\\^ + llgjll^) , 
hence from (fT2l) and the definition of ^(J), 

\\wj - wjII^ < |J|i/2^-i {^A{3) + \\qj\\J . 



(13) 



Thus, if we assume /i < ^ 



1/2 



AiJ) 



and 



I^jI 



< 



3|J|V2' 



we get 



\wj - wj||^ < 2z^/3, 



so that for all G G ^j, ||wg|Ioo — §> hence the hull is indeed selected. 
This also ensures that wj satisfies the equation (see Lemma [T4l) 



where 



Qjj {wj - wj) - gj + /ifj = 0, 

d'^ o o w 



(14) 



(15) 



(16) 



E 

Gegj 



\\d'^ o w\ 
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We now prove that the w padded with zeros on J is indeed optimal for the full problem with 
high probability. According to Lemma [141 since we have already proved (fT6l) . it suffices to show 
that 

{Q'jr[VL{w)j.]<fi. 
Defining qjc\j = qjc — Qji^jQjjQj, we can write the gradient of L on as 

VL(u))jc = - mQjcjQJjVj = - fiQjcjQjj{fj - rj) - fiQjcjQjjVj, 

which leads us to control the difference rj — rj. Using Lemma [121 we get 

II- II 11-^ II / IMjll2 \\d^ o d'^ o w\\'^ 

where w = tow + (1 — io)w for some to £ (0, 1). 
Let J = {A; G J : / 0} and let cp be defined as 

\\d^ o d^ o 

f= _ sup > 1. 

The term ip basically measures how close J and J are, i.e., how relevant the prior encoded by Q 
about the hull J is. By using (ITSl ). we have 



d*^ o ttill^ > \\dS- o w^W'i > \\d^ o dS-o w^Wi— > M'^ o d'^ o w\\i — , 

^iij JII2 JJ •'2) 6ip 

o w\\2 > \\dj o wj\\2 > Wdjh^ > \\dj 



3y^ 

and 



5 „ , 
kiloo < 3 l|w| 



Therefore we have 



i~ II . II . II / \\dj\\l , l|w||oo Mj o<ij||i 



\\d'^ o w\\2 V \\d'-^ o 'w\ 



2 



< 



31/^||^^J - wjII^ / 5(^||w 



1 + :^^^™^ ^ 11^, 



II2 



Introducing a = -^JwHoo J^Q^g^ , we thus have proved 

- rj||^ < Q ||u)j - wjII^ . (17) 

By writing the Schur complement of Q on the block matrices Qjcjc and Qjj, the positive- 
ness of Q implies that the diagonal terms diag(Qjcj(5j'j(5jjc) are less than one, which results in 
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— l/2ii 

{Qj^jQ jj lloo,2 < 1- We then have 



\Qj'^jQjj{fj - rj)| 



Qj-jQjj Qjj [rj - rj 



< IIQj^jQ7y'l|oo,2||Q"'/' 



JJ lloo,2 llt^jj' ||2 rj -rj||2 
O - rj||i 
< K-'/'a\J\'/Hf,A{J) + \\qj\U, 



(18) 

(19) 
(20) 
(21) 



where the last line comes from Eq. ( [T3l ) and dTTl ). We get 



(f^5)*[Qj^jQj](rj-rj)]< 



J 1 1/2 



^3/2a(Jc) 



-(M(J) + ||gj| 



Thus, if the following inequalities are verified 



a|J|i/M(J) r 
K3/2a(jc) - I' 

a|J|i/2 „ 



< 



At3/2a(J^) 



4' 
2 ' 



(22) 

(23) 
(24) 



we obtain 



i.e., J is exactly selected. 

Combined with earlier constraints, this leads to the first part of the desired proposition. 

We now need to make sure that the conditions ([T4l) . (l23l) and (l24l) hold with high probability. 
To this end, we upperbound, using Gaussian concentration inequalities, two tail-probabilities. First, 
gjc|j is a centered Gaussian random vector with covariance matrix 

IE[gjc|jgJc|j] = E qjcqjc - qj'^qj Qj^Qjj'^ - Qj'^jQjjqjqJc + Qj'^jQjjQjqjQjjQjj^ 



n 



-Q. 



j'^j^lj) 



where Qjcjc|j = Qjcjc — QjcjQjjQjjc In particular, (f2j)*[gjc|j] has the same distribution as 



tpCW), with ip : u ^ (r2j)*((Tri ^^^Qj^jcij^i) and W a centered Gaussian random variable with 



umt covariance matrix. 

Since for any u wehavejt7Qjcjc|jn < u^Qjc^cu < ||Qi/2||2 ||'"||2' by using Sudakov- 
Fernique inequality ( Adler . 199d. Theorem 2.9), we get: 

E[(J^S)*[9J1J] =IE sup n^(7j.|j < an-V2||g||V2E sup u'W 



n5(«)<i 



C5(«)<i 



<an-'i'\\Q\\yMm*m]- 
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In addition, we have 

i) - ^p{v)\ < ^Ij{u -v)< an-^/'^a{r)~^ <5icjc|j(^i - v) 



On the other hand, since Q has unit diagonal and QjcjQjjQjj^ has diagonal terms less than one, 

1 /2 

Qjcjc|j also has diagonal terms less than one, which implies that ||(5jcjc|j||oo,2 < 1- Hence ip is a. 

Lipschitz function with Lipschitz constant upper bounded by an~^/' ^a(J^)~\ Thus by concentra- 
tion of Lipschitz functions of multivariate standard random variables (lMassartl.l2003l. Theorem 3.4), 
we have for i > 0: 



\nmQj^\j]>t+^n-'^'\\Q\\Tnm*{w)] 



< exp 



2^ 



Applied for t = fiT/2 > 2an'^/'^\\Q\\l^\[{n''j)*{W)], we get (because (n - l)^ > ^2/4 for 
u > 2): 



^[{n'jr[qjc\j]>t] <exp 
It finally remains to control the term IP(||gj||oo > C)' with 



32cj2 



Kl^ 

? = — mm { 1, 



We can apply classical inequalities for standard random variables (|Massarti . l2003l . Theorem 3.4) that 
directly lead to 

2a2 J ■ 



kjlloo >0 <2|J|exp 
To conclude. Theorem |7] holds with 



and 



Ci(a,J) 

C2{g,j) 
C3(a,J) 

CAig,J) 



16 



min < 1, 



2V/'||w||ooEGegJl^j| 



4||Q||fE[(J]S)*(t^)] 



(25) 

(26) 
(27) 



3A{3) 



min < 1, 



2V/2||w|LEGegjKj1l2j ' 
where we recall the definitions: W a centered Gaussian random variable with unit covariance ma- 
trix, J = {j e J : Wj / 0}, v = min{|wj|; j G J}, 

lid'' o dP o u\\i 
'P= _ sup --, 

u&W:3'Z{k£3:uuT^0}C3 ll"j ° "j ° II 1 



K = Amm(Qjj) > and r > such that (r^j)*[(3jcj(3jjr] < 1 - r. 
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Appendix H. A first order approach to solve Eq. ^ and Eq. © 

Both regularized minimization problems Eq. Q and Eq. Q (that just differ in the s quaring of Q) can 



be so lved by using generic toolboxes for second-order cone programr ning (SQCP) (Bovd and Vandenberghd . 



2004 ). We propose here a first order approach that takes up ideas from BachI ( 2008b ) ; Micchelli and Pontil 



(|2006h and that is based on the following variational equalities: for x € W, we have 



mm 



V J2 



whose minimum is uniquely attained for = Similarly, we have 



2 ||x||i = mill — + 



li ' 



whose minimun is uniquely obtained for Zj = \xj\. Thus, we can equivalently rewrite Eq. (O as 

n p 

— y^J(yi,w~^xi) + - 



mill -Y^i(^y,,nj'^x^) + ^J2'^K' + ^\\('f^Geg\\i, (28) 



with (j = {J2G9j(^f)'^ (''1^)^^)^^ ■ same vein, Eq. ^ is equivalent to 

1 " A ^ 

S ;^E^(y--^-0 + 2E-.V' (29) 

where is defined as above. The reformulations Eq. (1281 ) and Eq. ( |29l ) lend themselves well to a 
simple alternating optimization scheme between w (for instance, t/; can be computed in closed-form 
when the square loss is used) and {r]^)Geg (whose optimal value is always a closed-form solution). 

This first order approach is computationally appealing since it allows warm-restart, which can 
dramatically speed up the computation over regulaiization paths. 

Appendix I. Technical lemmas 

In this last section of the appendix, we give several technical lemmas. We consider / C {1, . . . ,p} 
and Gi = {G e Q; Gnl ^ 0} C Q, i.e., the set of active groups when t he variables I are selected . 
We begin with a dual formulation of 0* obtained by conic duality ( Boyd and Vandenberghd . 



20041): 



Lemma 9 Let uj G Rl-'L We have 



(0/)*[n/] = mill max||^f||2 

s.t. Uj + dJCj = and (,f = if j ^G. 

Gegi,G3j 
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Proof By definiton of {Qj)*[uj], we have 

(Qr)*\ur] = max ujvr. 

Qi{vj)<l 

By introducing the primal variables {aG)Gegi ^ 1^'^^', we can rewrite the previous maximization 
problem as 

{Qi)*[ui]= max ujvj, s.t. yGeQi, \\df o UGni\\2 ^ o^g, 

which is a second-order cone program (SOCP) with \Qj\ second-order cone constraints. This pri- 
mal problem is convex and sati sfies Slater's conditions for gen eralized conic inequalities, which 
implies that strong duality holds ( Boyd and Vandenberghd . 2004 ). We now consider the Lagrangian 
jC defined as 

£{vi,aG,7,rG,^i) = uJvi+j{l-Y,(^G)+ Yl (jg "^^^ ) (Ig) ' 

with the dual variables {7, (,TG)Gegi,{S,i)Gegi} G M+ xRI^^^I xRI^I^'I^^I such that for all G G Gi, 
= if j ^ G and II2 < tg- The dual function is obtained by taking the derivatives of C 
with respect to the primal variables vj and (00)^661 ^^'^ equating them to zero, which leads to 

VjG/, Uj+ZGegr,G^,dfCf =0 

VGeg/, 7-TG =0. 

After simplifying the Lagrangian, the dual problem then reduces to 

^.^ fVjG/,n,+EGeg„G3,dfe/ = 0andC- = 0if j^G, 

which is equivalent to the displayed result. ■ 
Since we cannot compute in closed-form the solution of the previous optimization problem, we 



focus on a different but closely related problem, i.e., when we replace the objective max^gg^ 
by maxcgg^ ||^, to obtain a meaningful feasible point: 

Lemma 10 Letui e M'^L The following problem 



12 



m.m.,cG\ maxllff,,^ 



Gag, 

s.t. uj + df^f = and i° = Oifj^G, 

G€gi,GBj 



is minimized for {(,f)* — ^ 



J2Hej,Hegidj 
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Proof We proceed by contradiction. Let us assume there exists (Cf )gg6/ ^uch that 



maxll^fll^ < max II (^f, „^ 



max max 



YlHejo,Hegidjo ' 

where we denote by jo an argmax of the latter maximization. We notably have for all G 3 Jq: 

2^H&jo,Hegi"'jo 

By multiplying both sides by and by summing over G 3 jo, we get 



which leads to a contradiction. 



We now give an upperbound on 0* based on Lemma |9] and Lemma [TOl 
Lemma 11 Let uj G RI^L We have 



{^t)*\ut] < max < 
G&gi 



1 

2^ 2 



^H&iMGgr^f 



^H&j,H£gf^j 

Proof We simply plug the minimizer obtained in Lemma [10] into the problem of Lemma |9] ■ 

We now derive a lemma to control the difference of the gradient of j evaluated in two points: 

Lemma 12 Let uj, vj be two nonzero vectors in RI'^L Let us consider the mapping wj i— )• r{w j) = 

d'jod" 



Ylc&gj '\\dG^^ °^\\ ^ There exists zj = IqUj + (1 — to)vj for some to G (0, 1) such that 



\Gegj\\'^J°^-^h GGg.j ¥j°zj\\^ 

Proof For j, k£j,we have 
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with Ij=k = lif j = k and otherwise. We then consider t G [0, 1] i— hj (i) = rj (tuj + (1 — t)vj). 
The mapping hj being continuously differentiable, we can apply the mean-value theorem: there 
exists to G (0,1) such that 



h,(i) 



urn = 



We then have 



- rj{vj)\ < ^ 
fceJ 



< 



\uj - vj\ 



\Uk - Vk\ 



E 



id 



'j ) Fjl 



m°'42 t^JGT^j\\d3°ZJ 



which leads to 

\\r{uj) - r{vj)\\^ < \\uj - vj\ 



E 



\d^. 



J\\2 



\dj o dj o zj\\-^ 



GgGj \rJ ° ^-^h Gegj \\dj°zj 



Given an active set J C {1, . . . ,p} and a direct parent K e np(J) of J in the DAG of nonzero 
patterns, we have the following result: 

Lemma 13 For all G £ Gk\Gj, we have 

K\J C G 

Proof We proceed by contradiction. We assume there exists Go S Qk\Gj such that K\J ^ Go- 
Given that K e V, there exists G' C g verifying K = flcee' ^ ^' ^^^^^ 
definition GoHK ^ 0. 

We can now build the pattern K = Clceg'uiGo} G"^ = n Gq that belongs to V. Moreover, 
K = K CiGq C K since we assumed GqH K ^ 0.ln addition, we have that J C K and J C Gq 
because K £ I[-p{J) and Gq € Qk\Gj- This results in 

which is impossible by definition of K. ■ 

We give below an important Lemma to characterize the solutions of Q. 
Lemma 14 The vector w is a solution of 

min Liw) + uiliw) 

weRp 

if and only if 

VL{w) j + fifj = 

{n'jr[VL{w)j^]<^i, 
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with J the hull of {j G {1, . . . w)j 7^ 0} and the vector r defined as 

d'^ o dp o w 



E 



In addition, the solution w satisfies 

n*[VL{w)] < /X. 

Proof The problem 

min L{w) + fiQ{w) = min F{w) 



being convex, the directional derivative optimality condition are necessary and sufficient (IBorwein and Lewis , 



20061. Propositions 2.1.1-2.1.2). Therefore, the vector u; is a solution of the previous problem if and 



only if for all directions u &W,we have 



e>0 



Some algebra leads to the following equivalent formulation 



Vii G RP, u^VL{w) +fiu'jrj + fi [u j,] > 0. (30) 



The first part of the lemma then comes from the projections on J and J'^. 



An application of the Cauchy-Schwartz inequality on uTr ; gives for all n G M^" 



u]rj<{^j)[uj]. 
Combined with the equation (l30l ). we get 

V-u G u^VL{w) + fin{u) > 0, 
hence the second part of the lemma. ■ 

We end up w ith a lemma regarding the dual norm of the sum of two disjoint norms (see 
Rockafellai] . ll970h : 



Lemma 15 Let A and B be a partition of {1, . . . ,p}, i.e., ACi B = and AU B = {1, . . . ,p}. 
We consider two norms ua G M'"^' i— ?• HuaHa cind ub G RI'^I i— t- HubIIb. with dual norms \\va\\a 
and II'UbII^. We have 

max n'''T; = max II'^bIIb}- 

II"aIU+I1"sI1s<i 
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